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1 . THERE FOLLOWS PROPOSED US PAPER FOR CITED MEETING. 

2. QUOTE - INTRODUCTION: AT ITS MARCH 29-30, 1977 MEETING 

IN GENEVA, THE GENEVA GROUP CONSULTATIVE LEVEL DECIDED TO 
CONVENE A MEETING OF EXPERTS FROM THE RESPECTIVE CAPITALS 
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TO REVIEW AND MAKE RECOMMENDATIONS WITH RESPECT TO THE 
DEVELOPMENT OF A COMMON APPROACH TO BUDGET AND PROGRAM 
ANALYSIS AND EVALUATION, WHICH COULD BE USED IN GENEVA 
GROUP CAPITALS TO FACILITATE EACH COUNTRY'S REVIEW OF THESE 
ORGANIZATIONS' BUDGET AND PROGRAM PROPOSALS. THE EXPERTS 
MEETING, HELD IN GENEVA ON JULY 7-8, 1977, PRODUCED A 
DOCUMENT ENTITLED "STANDARD ANALYTICAL FRAMEWORK FOR PROGRAM 
BUDGET ANALYSIS," WHICH WAS APPROVED BY THE GENEVA GROUP 
CONSULTATIVE LEVEL AT ITS OCTOBER 6-7, 1977 MEETING IN NEW 
YORK. THE SAF OUTLINES THE COMMON ELEMENTS OF BUDGETARY 
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ANALYSIS, PROVIDES GENERAL INSTRUCTIONS FOR GENEVA GROUP 
TECHNICAL REVIEW OF SPECIALIZED AGENCY BUDGET DOCUMENTS, 
AND REQUESTS THE SPECIALIZED AGENCIES TO PROVIDE ADDITIONAL 

AND/OR STANDARDIZED BUDGETARY INFORMATION IN ORDER TO 
FACILITATE NATIONAL REVIEW OF THESE DOCUMENTS. THE 
EXPERTS, HOWEVER, WERE UNABLE DUE TO TIME CONSTRAINTS TO 
ADDRESS THE BROADER AND MORE COMPLEX QUESTION OF PROGRAM 
ANALYSIS, EVALUATION AND REVIEW AT ITS JULY 1977 MEETING. 

THE GENEVA GROUP CONSULTATIVE LEVEL THEREFORE AGREED TO 
CONVENE ON FOLLOW-UP EXPERTS MEETING TO CONSIDER THIS 
SUBJECT AND, IF POSSIBLE, TO AGREE UPON A COMMON APPROACH 
TO THE REVIEW OF SPECIALIZED AGENCY PROGRAMS WHICH WOULD 
COMPLEMENT THE SAF DEVELOPED BY THE EXPERTS AND APPROVED 
BY THE CONSULTATIVE LEVEL GROUP. THE PURPOSE OF THIS 
PAPER IS TO PROVIDE A BASIS FOR DISCUSSION AND A POSSIBLE 
STRUCTURE FOR DEVELOPING SUCH A COMMON APPROACH. 

3. PART 1 DISCUSSES THE PURPOSE OF THE EXERCISE, OUTLINES 
POSSIBLE CONSTRAINTS AGAINST EFFECTIVE PROGRAM REVIEW OF 
THE SPECIALIZED AGENCIES, DELINEATES BETWEEN PROGRAM 
ANALYSIS, EVALUATION AND REVIEW, AND ATTEMPTS TO REALIS- 
TICALLY DETERMINE WHAT SHOULD OR SHOULD NOT BE EXPECTED OF 
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GENEVA GROUP ANALYSTS IN REVIEWING THESE PROGRAMS. PART 
II OUTLINES THE BASIC ELEMENTS OF PROGRAM REVIEW, THE 
VARIOUS METHODOLOGICAL APPROACHES TO PROGRAM REVIEW IN 
CAPITALS, AND DISCUSSES THE PROCESS OF COORDINATION 
REQUIRED FOR SUCH REVIEW BOTH WITHIN CAPITALS AND WITHIN 
THE GENEVA GROUP STRUCTURE. PART III OFFERS RECOMMENDA- 
TIONS WHICH THE GENEVA GROUP MAY WISH TO CONVEY TO THE 
SECRETARIATS WITH RESPECT TO IMPROVING THEIR OWN PROGRAM 
ANALYSIS AND EVALUATION CAPABILITIES AND, CONSEQUENTLY, 
FACILITATING GENEVA GROUP PROGRAM REVIEW. 

4. ON DEVELOPING A COMMON APPROACH TO THE REVIEW OF 
SPECIALIZED AGENCY PROGRAMS. 

5. I. PURPOSE AND EXPECTATIONS OF A GENEVA GROUP PROGRAM 
REVIEW EXERCISE. 

6. OVER THE COURSE OF THE LAST SEVERAL YEARS, THE GENEVA 
GROUP HAS BECOME INCREASINGLY AWARE OF THE NEED FOR THE 
MAJOR CONTRIBUTORS TO MAKE A MORE EXERTED AND COORDINATED 
EFFORT AT REVIEWING THE PROGRAMS OF THE UN SPECIALIZED 
AGENCIES AND RELAYING PROGRAM RECOMMENDATIONS EMANATING 
FROM THESE REVIEWS TO THE VARIOUS SECRETARIATS AT AN EARLY 
STAGE IN THE PROGRAM DECISIONS OF THE SECRETARIATS. BY 
IMPROVING THE CAPABILITIES OF THE GENEVA GROUP MEMBERS IN 
REVIEWING SPECIALIZED AGENCY PROGRAMS. WE WILL BE IN A 
BETTER POSITION TO: 
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- RECOMMEND CHANGES IN PROGRAM EMPHASIS OR DIRECTION 
WHICH REFLECT OUR COLLECTIVE JUDGMENTS AS TO THE BEST 
POSSIBLE DISTRIBUTION OF RESOURCES; 

- IDENTIFY PROGRAMS WHICH HAVE OUTLIVED THEIR USEFULNESS 
OR ARE OF A LOWER PRIORITY COMPARED TO OTHER DEMANDS ON 
RESOURCES; 

INDICATE WHICH PROGRAMS APPEAR TO BE FALLING SHORT OF 
MEETING STATED OBJECTIVES AND THAT SHOULD BE EVALUATED OR 
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RE-EXAMINED; AND 

-IDENTIFY PROGRAMS WHICH MAY BE DUPLICATIVE OF OTHER 
EFFORTS. 

7. HOWEVER, IT SHOULD BE MADE CLEAR AT THE BEGINNING THAT 
EFFECTIVE PROGRAM REVIEW BY THE GENEVA GROUP IS COMPLI 
CATED BY A NUMBER OF FACTORS: 

- PROGRAM ANALYSIS, EVALUATION AND REVIEW REMAIN VERY 
INEXACT SCIENCES AND NO TRULY OBJECTIVE METHODS OF JUDGING 
PRIORITIES AMONG PROGRAMS OR EFFECTIVENESS OF PROGRAMS 
HAVE BEEN DISCOVERED; 

- MOST GENEVA GROUP ANALYSTS ARE UNFAMILIAR WITH AN 
ORGANIZATION'S PROJECTS, SUB-PROGRAM, AND OFTEN EVEN MAJOR 
PROGRAMS, DUE PRIMARILY TO INADEQUATE DATA AND THE REL- 
ATIVE LACK OF EMPHASIS ON ANALYSIS AND REVIEW OF THESE 
PROGRAMS; 

- THE ORGANIZATIONS HAVE FAILED TO INCORPORATE PERFORMANCE 
AND EVALUATION CRITERIA INTO THEIR PROGRAM DEVELOPMENT, 
MAKING IT VERY DIFFICULT FOR ANALYSTS TO MEASURE PROGRAM 
EFFECTIVENESS; 

PROGRAM OBJECTIVES ARE OFTEN DEFINED TOO BROADLY TO BE 
ABLE TO RELATE PROGRAM ACTIVITY TO THE FULFILLING OF 
THOSE OBJECTIVES; 

- MOST ORGANIZATIONS’ PROGRAMS, WHETHER AT HEADQUARTERS 
OR IN THE FIELD, ARE FAR REMOVED PHYSICALLY FROM GENEVA 
GROUP COUNTRIES AND THUS CONSIDERABLE EXPENSE IS INCURRED 
IN CARRYING OUT EXAMINATIONS OF HEADQUARTERS OPERATIONS OR 
FIELD EVALUATIONS; 

- POLITICAL CONSIDERATIONS, SUCH AS AN UNWILLINGNESS TO 
APPEAR HOSTILE TO DEVELOPING COUNTRIES INTERESTS AND 
DEMANDS, MAY UNDERCUT MUCH HARD ANALYTICAL WORK AND 
EVENTUALLY MAKE THE EFFORT SEEM WASTED; 

-DIFFICULTY WILL BE ENCOUNTERED IN ATTEMPTING TO COORDI 
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NATE THE REVIEW AMONG THE VARIOUS DOMESTIC AGENCIES IN 
CAPITALS WHICH HAVE CLIENTELE INTERESTS IN SOME PART OF AN 
ORGANIZATION'S ACTIVITIES; 

IT WILL BE DIFFICULT TO DEVELOP A UNIFIED GENEVA GROUP 
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POSITION OF AN ORGANIZATION S PROGRAM IN A TIMELY ENOUGH 
MANNER TO INFLUENCE DECISION MAKING. 

8. THESE FACTOR. HOWEVER, REFLECT THE REALITIES OF 
OPERATING IN THE MULTILATERAL ARENA AND SHOULD NOT BE 
VIEWED AS EXCUSES AGAINST ENGAGING IN A PROGRAM REVIEW 
EXERCISE. RATHER, THESE CONSTRAINTS SHOULD BE CONSIDERED 

IN DETERMING REALISTICALLY WHAT SHOULD OR SHOULD NOT BE 
EXPECTED OF ANALYSTS, WHAT TYPE OF "PRODUCT" CAN BE 
DEVELOPED FROM SUCH A REVIEW, WHAT TYPE OF APPROACH WILL 
MOST EFFECTIVELY ACHIEVE THE EXPECTED RESULTS, AND HOW 
THE AGENCIES THEMSELVES CAN IMPROVE THEIR INTERNAL 
ANALYSIS AND EVALUATION CAPABILITIES AND THUS MAKE OUR 
TASK LESS DIFFICULT. 

9. PROGRAM ANALYSIS VS. PROGRAM EVALUATION VS. PROGRAM 
REVIEW. 

10. BEFORE PROCEEDING TO DISCUSS A COMMON APPROACH TO 
REVIEWING SPECIALIZED AGENCY PROGRAMS, IT IS IMPORTANT TO 
CONCEPTUALLY CLARIFY THESE TERMS SO AS TO PUT US ON COMMON 
GROUND WHEN DISCUSSING THESE ACTIVITIES. 

11. PROGRAM ANALYSIS IS ESSENTIALLY THE SYSTEMATIC, 

EXPLICIT EXAMINATION OF ALTERNATIVE WAYS TO REACH POLICY 
OBJECTIVES. IT INCLUDES ASSISTING THE POLICYMAKER IN 
ESTIMATING THE DESIRABILITY AND FEASIBILITY OF DIFFERENT 
STRATEGIES OF POLICY IMPELEMENTATION, AND INVOLVES ESTI- 
MATING FUTURE COSTS, EFFECTIVENESS, AND ANY OTHER SIGNI- 
CANT IMPACTS OF EACH ALTERNATIVE. PROGRAM ANALYSIS IS 
ESSENTIALLY FOCUSED ON FUTURE ACTIVITIES OF AN ORGANI 
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ZATION. PROGRAM EVALUATION, ON THE OTHER HAND, ASSESSES 
THE PAST PERFORMANCE OF EXISTING PROGRAMS IN ACHIEVING 
THEIR OBJECTIVES AND AIMS AT PROGRAM IMPROVEMENT THROUGH A 
MODIFICATION OF CURRENT OPERATIONS. THE FINDINGS OF 
PROGRAM EVALUATIONS PROVIDE IMPORTANT INFORMATION WHICH 
IS NEEDED FOR PROGRAM ANALYSIS, SINCE PROGRAM ANALYSIS 
NORMALLY CONSIDERS AN EXISTING PROGRAM TO BE ONE OF THE 
ALTERNATIVES TO BE EXAMINED. THUS, EVALUATION'S PREOC- 
CUPATION WITH EXISTING PROGRAMS DIFFERS FROM PROGRAM 
ANALYSIS, WHICH USUALLY COMPARES EXISTING AND HYPOTHETICAL 
ALTERNATIVE PROGRAM SOLUTIONS TO THE SAME PROBLEM. 

12. BOTH PROGRAM ANALYSIS AND PROGRAM EVALUATION USUALLY 
APPLY TO PROCESSES WITHIN AN ORGANIZATION AS OPPOSED TO 
INDEPENDENT EXTERNAL PROCESSES. THE PROGRAM ANALYST, IN 
THIS SENSE, IS ONE WHO IS CLOSELY ASSOCIATED WITH A PRO- 
GRAM AND IS IN A POSITION TO RECOMMEND TO POLICYMAKERS 
SEVERAL ALTERNATIVE COURSES OF ACTION FOR MEETING AN 
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OBJECTIVE. IN THE SAME SENSE, THE EVALUATOR IS NORMALLY 
OPERATING FROM WITHIN THE PLANNING AND PROGRAMING 
STRUCTURE OF THE ORGANIZATION (OR FROM A SYSTEM-WIDE UNIT 
SUCH AS THE JIU), RELAYING OBSERVATIONS TO THE ANALYST, 

WHO IN TURN WEIGHS THE CURRENT OPERATION AGAINST OTHER 
POSSIBLE ALTERNATIVES. THUS. IT IS PERHAPS INAPPROPRIATE 
TO USE THE TERMS ' PROGRAM ANALYSIS" AND "PROGRAM 

EVALUATION" TO REFER THE PROCESS WE HOPE TO DEVELOP WITHIN 
THE GENEVA GROUP. SINCE GENEVA GROUP ANALYST ARE ESSEN- 
TIALLY ON THE OUTSIDE LOOKING IN - REACTING TO PROGRAM 
PROPOSALS RATHER THAN FORMULATING ALTERNATIVES OR WEIGH- 
ING PROGRAM OPTIONS - IT SHOULD BE OBVIOUS THAT WE CANNOT 
EXPECT TO COMPETE WITH THE INTERNAL ANALYTICAL AND EVALU- 
ATION CAPABILITIES OF THE ORGANIZATIONS IN ANY DEGREE OF 
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PROGRAM DETAIL, SHORT OF CHALLENGING A SELECTED FEW 
ACTIVITIES. IN ESSENCE, WE NEED A MORE DESCRIPTIVE WORD 
TO USE IN REFERRING TO OUR EXTERNAL PROCESS. 

13. THE WORK "REVIEW" MORE ACCURATELY DESCRIBES A PROCESS 
WHICH CAN BE DEVELOPED WITHIN CAPITALS AND REFINED TO 
WHATEVER LEVEL OF SOPHISTICATION DEEMED DESIRABLE WITHIN 
THE CAPABILITIES OF THE RESPECTIVE GENEVA GROUP COUNTRIES. 
REVIEW IMPLIES A BROADER EXAMINATION OF WHERE AN ORGANI- 
ZATION IS AT A PARTICULAR TIME AND WHERE IT SHOULD BE 
GOING, WITHIN THE CONTEXT OF LOOKING AT IT FROM THE 
OUTSIDE AND THE IMPORTANCE ATTACHED TO THE ORGANIZATION 
AS A WHOLE AND TO SPECIFIC PROGRAM EMPHASES OF THE ORGAN- 
IZATION. REVIEW ALSO IMPLIES A LESS ANALYTICAL, LESS 
OBJECTIVE PROCESS THAN PROGRAM ANALYSIS OR EVALUATION, 
ALTHOUGH THE ELEMENTS OF THESE PROCESSES WILL CERTAINLY 
BE APPLIED TO SOME EXTENT IN THE REVIEW PROCESS. THIS IS 
NOT BAD IN AND OF ITSELF - IT IS MERELY A REALIZATION 

THAT OUR CAPABILITIES, AT LEAST IN THE NEAR TERM, ARE 
LIMITED. AND IT FURTHER CALLS INTO QUESTION WHETHER IT 
IS THE APPROPRIATE ROLE OF MEMBER STATES TO PERFORM THE 
FUNCTIONS OF THE SECRETARIATS. FURTHERMORE, BY AVOIDING 
THE USE OF THE WORD "EVALUATION" IN REFERRING TO OUR 
PROCESS, WE WILL BE ABLE TO PROTECT THE INTEGRITY OF THE 
JOINT INSPECTION UNIT AS IT ATTEMPTS TO STRENGTHEN ITS 
ROLE IN UN-SYSTEM-WIDE EVALUATION ACTIVITIES. 

14. REASONABLE EXPECTATIONS 

15. SINCE THIS PAPER HAS ALREADY ASSERTED THAT A NUMBER 
OF FACTORS CONSTRAIN THE REVIEW OF SPECIALIZED AGENCY 
PROGRAMS, AND THAT SUCH A REVIEW COULD IN NO WAY COMPETE 
WITH THE INTERNAL PROCESS OF ANALYSIS AND EVALUATION WITHIN 
THE ORGANIZATIONS, A VALID QUESTION BECOMES WHAT SHOULD 
WE EXPECT OR NOT EXPECT FROM THIS EXERCISE"? 
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16. BEFORE ATTEMPTING TO ANSWER THE QUESTION DIRECTLY, 

TWO ASSUMPTIONS SHOULD BE TABLED: 

- ALTHOUGH MANPOWER AND RESOURCE CAPABILITIES VARY AMONG 
GENEVA GROUP GOVERNMENTS, THERE MUST BE SOME MINIMUM 
STANDARD OF PROGRAM REVIEW ACTIVITY PERFORMED BY EACH 
GOVERNMENT IS THE UNIFIED APPROACH IS TO SUCCEED. 

- LIKE ANY NEW PROCESS, EXPECTATIONS SHOULD NOT BE SET 
TOO HIGH IN THE FIRST YEARS. PERHAPS GOVERNMENTS COULD 
AGREE TO PERFORM PRIORITY RANKINGS OF PROGRAMS IN THE FIRST 
YEAR, AND THEN ATTEMPT TREND ANALYSIS OR INPUT/OUTPUT OR 
OTHER MORE REFINED ANALYSES IN SUBSEQUENT YEARS. THIS, 

OF COURSE, IS CONTINGENT UPON THE ABILITY OF EACH GOVERN- 
MENT TO DEVELOP AN INTERNAL REVIEW PROCESS WHICH CAN 
DEVELOP MORE ANALYTICAL EXPERTISE OVER TIME. 

17. IN GENERAL, REASONABLE EXPECTATIONS FOR SUCH AN 
EXERCISE WOULD INCLUDE THE FOLLOWING: 

(1) EACH GENEVA GROUP GOVERNMENT SHOULD BE ABLE TO 
DEVELOP AN INTERNAL PROCESS (OR IMPROVE AN EXISTING 
PROCESS) FOR THE TIMELY REVIEW OF SPECIALIZED AGENCY 
PROGRAMS. 

(2) GENEVA GROUP ANALYSTS SHOULD BE CAPABLE OF PROVIDING 
IN A SHORT PERIOD OF TIME A PRIORITY RANKING OF MAJOR 
PROGRAMS, AND PERHAPS A RANKING OF SUB-PROGRAMS WITHIN 
EACH PROGRAM. 

(3) GENEVA GROUP ANALYSTS WHOULD BE ABLE TO IDENTIFY 
PROGRAMS WHICH RAISE DOUBT AS TO THEIR CONTINUED USE- 
FULNESS, MARGINAL BENEFIT, OR VALUE AS PERCEIVED BY THE 
GOVERNMENTS. 

(4) THE GENEVA GROUP SHOULD BE ABLE TO AGREE UPON SOME 
GENERAL PROGRAM DIRECTIONS STEMMING FROM THE PRIORITY 
RANKING PROCESS, AND TO AGREE ON THOSE ACTIVITIES WHICH 
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SHOULD BE RE-EXAMINED OR EVALUATED, EITHER BY THE GENEVA 
GROUP OR THE JIU. 

(5) THE INTERNAL AND GENEVA GROUP PROCESSES SHOULD BE 
ABLE TO BE DEVELOPED WITHIN A TIMEFRAME WHICH ALLOWS RE- 
COMMENDATIONS TO BE PASSED ON TO THE SECRETARIATS AT AN 
EARLY STAGE IN THEIR PROGRAM AND BUDGET CYCLES. 

(6) GENEVA GROUP GOVERNMENTS SHOULD BE ABLE TO DEVOTE 
ADDITIONAL RESOURCES AS NEEDED TO CARRY OUT EVALUATIONS 
OF SELECTED PROGRAMS IDENTIFIED IN THE REVIEW PROCESS. 

(7) EVALUATIONS SHOULD BE IN A FORM WHICH CAN BE PASSED 
ALONG TO OTHER GENEVA GROUP GOVERNMENTS AND TO THE 
SECRETARIATS. 
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18. UNREASONABLE EXPECTATIONS WOULD INCLUDE THE FOLLOWING: 

(1) PROGRAM REVIEWS IN CAPITALS SHOULD NOT BE FOCUSED 

IN SUCH DETAIL SO AS TO CLOG THE PROCESS WITH UNNECESSARY 
REVIEW. FURTHERMORE, THE VALUE OF SUCH DETAILED REVIEW 
IS OF MARGINAL BENEFIT UNLESS THE GENEVA GROUP CAN SYNTHE- 
SIZE AND AGREE UPON SPECIFIC PROGRAMMATIC DETAILS (WHICH 
IS UNLIKELY IN THE NEAR FUTURE). 

(2) PROGRAM REVIEWS PROBABLY CANNOT BE BACKED UP WITH ANY 
APPRECIABLE DEGREE OF PROGRAM ALALYSIS, SINCE GENEVA GROUP 
ANALYSTS WILL HAVE INADEQUATE DATA FROM WHICH TO WORK IN 

THE FIRST PLACE. THIS WILL REPRESENT SOMEWHAT OF A 
PROBLEM WHEN PROGRAM RECOMMENDATIONS ARE PASSED ON TO THE 
SECRETARIATS, WHO ASK FOR SPECIFIC PROGRAMMATIC DETAILS 
KNOWING FULL WELL THAT OUR ABILITIES TO PRODUCE SUCH 
DETAILS ARE LIMITED. 

(3) COORDINATION OF PROGRAM RECOMMENDATIONS AMONG THE 
GENEVA GROUP BECOMES MORE DIFFICULT IN PROPORTION TO THE 
AMOUNT OF PROGRAM DETAIL GENERATED BY INTERNAL REVIEWS, 

AS STATED ABOVE. TO EXPECT A UNIFIED GENEVA GROUP 
"POSITION" TO DEVELOP IN THE INITIAL ATTEMPT IS PERHAPS 
UNREALISTIC. ALTHOUGH THE BENEFITS OF THE INTERGOVERN- 
MENTAL PROCESS WILL BE WORTHWHILE. 
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19. PART II OF THIS PAPER ATTEMPTS TO DEVELOP A COMMON 
APPROACH TO PROGRAM REVIEW BASED UPON THESE EXPECTATIONS 
AND LIMITATIONS. 

20. II. COMMON APPROACH TO THE REVIEW OF SPECIALIZED 
AGENCY PROGRAMS. 

21. THIS SECTION ELUCIDATES THE BASIC ELEMENTS OF PROGRAM 
REVIEW AND RELATED THEM TO SPECIALIZED AGENCY PROGRAMS. 
OUTLINES VARIOUS METHODOLOGICAL APPROACHES TO PROGRAM 
REVIEW IN CAPITALS, AND DISCUSSES THE PROCESS OF COORDI- 
NATION REQUIRED FOR SUCH REVIEW BOTH WITHIN CAPITALS AND 
AMONG THE RESPECTIVE GENEVA GROUP COUNTRIES. 

22. BASIC ELEMENTS OF PROGRAM REIVEW 

23. THE FOLLOWING SERIES OF QUESTIONS INCORPORATE THE 
BASIC ELEMENTS OF PROGRAM REVIEW: 

- WHAT IS THE PROBLEM? WHAT ARE THE CAUSES? TO WHAT 
EXTENT ARE THEY KNOWN? WHAT CLIENT GROUPS ARE AFFECTED. 
WHAT IS THE MAGNITUDE OF THE PROBLEM? 

- TOWARD WHAT OBJECTIVES SHOULD PROGRAMS FOR MEETING 
THE PROBLEM BE DIRECTED, I.E., FUNDAMENTAL PURPOSES? 

- ARE PROGRAM OBJECTIVES WELL-DEFINED? ARE THE OBJECTIVES 
OF ON-GOING PROGRAMS STILL VALID? 

- HOW LONG HAS THE PROGRAM BEEN UNDERWAY? WHEN WAS IT 
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LAST REVIEWED BY A POLICYMAKING ORGAN? WHEN IS IT EX- 
PECTED TO BE COMPLETED? 

- WHAT EVALUATION CRITERIA ARE BUILT INTO THE PROGRAM? 

SHORT OF THESE CRITERIA, HOW CAN ESTIMATES OF PROGRESS 
AGAINST OBJECTIVES BE MADE. 

- WHAT OTHER AGENCIES ARE ATTEMPTING TO MEET THE PROBLEM? 
UNCLASSIFIED 
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IS THERE DUPLICATION OF EFFORT WITH THESE AGENCIES? 

- WHAT IS THE DEGREE OF POLITICAL SUPPORT BEHIND THE 
PROGRAM? 

- WHAT ALTERNATIVE PROGRAMS OR ACTIVITIES SHOULD BE 

CONSIDERED? 

24. ANSWERS TO THESE QUESTIONS SHOULD ENABLE THE ANALYST 
TO PROCEED TO CONSIDERATION OF FOUR CATEGORIES OF CRITERIA 
BY WHICH THE SUCCESS OR FAILURE OF A PROGRAM MAY BE 
JUDGED: 

(1) EFFORT - THIS IS AN ASSESSMENT OF INPUT OR WORKLOAD 
WITHOUT REGARD TO OUTPUT. HERE THE CRITERION OF SUCCESS 
IS THE QUANTITY AND QUALITY OF ACTIVITY THAT TAKES PLACE. 

(2) EFFECTIVENESS - THIS IS A PERFORMANCE CRITERION 
MEASURING THE RESULTS OF EFFORT RATHER THAN THE EFFORT 
ITSELF AND REQUIRES A CLEAR STATEMENT OF OBJECTIVES. 

(3) IMPACT - HERE THE CRITERION OF SUCCESS IS THE DEGREE 
TO WHICH EFFECTIVE PERFORMANCE IS ADEQUATE TO THE TOTAL 
AMOUNT OF NEED. 

(4) COST EFFECTIVENESS - THIS CRITERION REPRESENTS A 

RATIO BETWEEN EFFORT AND IMPACT AND ASKS WHETHER THE COST 
IS JUSTIFIED BY THE IMPACT MADE BY THE PROGRAM. 

25. THESE CRITERIA WILL BE DISCUSSED MORE EXPLICITY IN THE 
FOLLOWING SECTION RELATING TO THE NATURE OF THE PROGRAMS TO 
BE REVIEWED. 

26. THE NATURE OF SPECIALIZED AGENCY PROGRAMS. 

27. APPROXIMATELY 70 OF THE REGULAR BUDGETS OF THE 
SPECIALIZED AGENCIES ARE COMPRISED OF PERSONNEL COSTS. 

THUS. AGENCY REGULAR ''PROGRAMS" ESSENTIALLY INCLUDE 
STUDIES, MEETINGS AND CONFERENCES, COLLECTION AND DISSEMI- 
NATION OF STATISTICS AND INFORMATION, AND RELATED BUREAU- 
CRATIC ACTIVITIES. IN MANY CASES, HOWEVER, THE MAJOR 
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PROGRAM IS A COMBINATION OF REGULAR BUDGET AND EXTRA- 
BUDGETARY ACTIVITIES WHICH ARE DIFFICULT TO DISTINGUISH 
IN THE NARRATIVE SECTIONS OF THE PROGRAM BUDGET DOCUMENTS. 
IT IS THUS DIFFICULT FOR THE REVIEWER TO DETERMINE THE 
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ADEQUACY OF REGULAR PROGRAMMATIC INPUTS AND THE IMPACT 
OF THOSE RESOURCES IN FULFILLING PROGRAM OBJECTIVES. BUT 
THE REAL QUESTION REMAINS, HOW SHOULD THE REVIEWER PROCEED? 

28. FIRST, THE REVIEWER MUST ASK, TO THE BEST OF HIS 
ABILITY, WHETHER THE EFFORT IS WORTHWHILE. GIVEN THE 
MAGNITUDE OF PROBLEMS FACED AND THE GROWING REALIZATIONS 
OF RESOURCE LIMITATIONS. THE QUESTION BECOMES FAIR, BOTH 
INTELLECTUALLY AND IN PRACTICAL TERMS. OF COURSE, THIS 
REPRESENTS THE MOST COMPLEX QUESTION ASKED IN THE REALM 
OF PUBLIC POLICY, AND THE ANSWER CAN ONLY BE SUBJECTIVE. 

THE USUAL RESPONSE OF POLICYMAKERS IS TO THROW A COM- 
PARATIVELY SMALL AMOUNT OF RESOURCES AT A WIDE VARIETY OF 

PROBLEMS, RATHER THAN FOCUSING IN ON THE MOST CRITICAL 
ONES. THIS PHENOMENON IS EVEN GREATER IN INTERNATIONAL 
ORGANIZATIONS. WHERE SECRETARIATS HAVE LESS LATITUDE IN 
DETERMINING THE PRIORTIZATION OF PROGRAMS. BUT SHORT OF 
ASKING THE ORGANIZATIONS TO CUT BACK PROGRAMS BY A CERTAIN 
AMOUNT WITHOUT PROVIDING ANY RECOMMENDATIONS AS TO WHERE 
THE REDUCTIONS WOULD BE THE MOST APPROPRIATE, THIS 
QUESTION OF APPROPRIATE EFFORT MUST BE ADDRESSED. 

29. SECOND, THE REVIEWER MUST MAKE AN ATTEMPT TO JUDGE 
THE EFFECTIVENESS OF A PROGRAM, OR AT LEAST TO CONCLUDE 
THAT THE EFFECTIVENESS MAY BE LESS THAN OPTIMAL. THIS IS 
ATTEMPTED BY LAYING OUT THE OBJECTIVES AND MATCHING THE 
PROGRAM INPUTS AGAINST THE OBJECTIVES. THIS IS DIFFICULT 
WITHOUT SOME TYPE OF EVALUATION MECHANISM. BUT THE TASK 
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CAN BE MADE EASIER BY THE INCORPORATION OF PERFORMANCE 
CRITERIA INTO THE DESIGN AND IMPLEMENTATION OF THE PROGRAM. 

30. THIRD. THE IMPACT OF THE PROGRAMS MUST BE SCRUTINIZED. 

A PROGRAM MAY BE WELL CONCEIVED AND EFFECTIVE IN MEETING 
ITS OBJECTIVES, BUT MAY MAKE ONLY A SLIGHT CONTRIBUTION 
RELATIVE TO THE TOTAL AMOUNT OF NEED. A LARGER POPULATION 
MAY BE SERVED BY APPLYING THE RESOURCES IN A DIFFERENT 
FASHION. 

3 1 . FOURTH, COST HAS TO BE A CRITICAL ELEMENT IN ANY 
REVIEW. DETERMINING THE COST EFFECTIVENESS OF A SPECIAL- 
IZED AGENCY PROGRAM WILL ALSO BE DIFFICULT. SINCE QUANTI- 
FIABLE MEASUREMENTS OF BENEFITS SELDOM EXIST. ONCE 
AGAIN, SUBJECTIVE JUDGMENTS WILL HAVE TO BE RELIED UPON. 

IF THE IMPACT IS DETERMINED TO BE LESS THAN THE RELATIVE 
COST. THIS JUDGMENT SHOULD BE MADE KNOWN TO THE GENEVA 
GROUP AND THE SECRETARIATS. 

32. OBVIOUSLY THE FOREGOING DISCUSSION IS "BOILERPLATE" 

AND CAN BE FOUND IN ALMOST EVERY TEXTBOOK ON PROGRAM 
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ANALYSIS OR PUBLIC POLICY DEVELOPMENT. THE POINT THAT 
SHOULD BE MADE IS THAT ONCE A PROCESS GETS UNDERWAY - 
ONCE GENEVA GROUP ANALYSTS AND REPRESENTATIVES BEGIN TO 
LOOK SERIOUSLY AT THE PROGRAMS OF THESE AGENCIES - THESE 
GENERAL QUESTIONS AND CRITERIA WILL BRING TO LIGHT A NUM- 
BER OF SUBJECTIVE JUDGMENTS AND OBSERVATIONS WHICH CAN BE 
ADDRESSED COLLECTIVELY BY THE GENEVA GROUP. 

33. VARIOUS METHODOLOGICAL APPROACHES TO SPECIALIZED AGENC 
PROGRAM REVIEW. 

34. AS MENTIONED PREVIOUSLY, THE DEVELOPMENT OF A GENEVA 
GROUP REVIEW PROCESS MUST BEGIN WITH A RELATIVELY SIMPLE 

REFINED AS THE GENEVA GROUP DESIRES. KEEPING IN MIND 
UNCLASSIFIED 
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OUR EARLIER DISCUSSIONS, THIS SECTION PROPOSES THREE 
METHODOLOGIES, IN THE ORDER OF THEIR PROBABLE EVOLUTION, 
WHICH COULD BE UTILIZED AS DEEMED APPROPRIATE IN GENEVA 
GROUP CAPITALS. 

(1) PRIORITY RANKING - THE SIMPLEST FORM OF PROGRAM 
REVIEW INVOLVES THE RANK ORDERING OF THE VARIOUS AGENCY 
PROGRAMS ACCORDING TO EACH MEMBER GOVERNMENT S PERCEPTION 
OF THEIR IMPORTANCE RELATIVE TO OTHER PROGRAMS CARRIED 
OUT BY THE AGENCY. PRIORITY RANKING AT THE SUB-PROGRAM 
LEVEL (PROBABLY THE MOST VALID IN THE EARLY STAGES OF THE 
GENEVA GROUP PROCESS) SHOULD NOT REPRESENT MAJOR DIF 
FICULTIES, BECAUSE (A) MOST SPECIALIZED AGENCY BUDGET 
DOCUMENTS ARE DEVELOPED ALONG THESE LINES, AND (B) THE 
MINISTRIES SHOULD BE ABLE TO PRIORITIZE AMONG ACTIVITIES 
CARRIED OUT IN THE SAME NARROW SECTOR. FOR EXAMPLE, FAO'S 
PROGRAM OF FISHERIES EXPLOITATION AND UTILIZATION IS 
COMPOSED OF SIX SUBPROGRAMS: MARINE RESOURCES AND 
ENVIRONMENT, INLAND WATER RESOURCES, FISH PRODUCTION, 

FISH UTILIZATION, REGIONAL OFFICES, AND FIELD PROGRAM. 

A CAREFUL READING OF THE BUDGET DOCUMENTS, AS WELL AS 
OTHER INFORMATION GAINED THROUGH PARTICIPATION IN MEETINGS 
AND CONFERENCES RELATING TO THE VARIOUS SUBPROGRAMS, 
SHOULD PROVIDE AN INDICATION, ALBEIT SKETCHY, OF WHAT THE 
ORGANIZATION IS DOING IN THIS PROGRAM AREA. CONCLUSIONS 
CAN THEN BE REACHED AMONG THE EXPERTS IN CAPITALS AS TO 
THE RELATIVE IMPORTANCE OF EACH SUBPROGRAM IN THE AREA OF 
FISHERIES EXPLOITATION AND UTILIZATION. PRIORITY RANKING 
BECOMES MORE DIFFICULT AT THE PROGRAM LEVEL, WHERE FISH 
ERIES IS RANKED AGAINST FORESTRY OR AGRICULTURE. BUT THE 
IMPORTANT POINT TO BE MADE HERE IS THAT SUCH RANKING, 
REGARDLESS OF THE DEGREE OF SUBJECTIVITY USED, IS A 
REALISTIC STARTING POINT FOR THE DEVELOPMENT OF A GENEVA 
GROUP PROGRAM REVIEW PROCESS. SUCH A METHODOLOGY WOULD 
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ENABLE GOVERNMENTS TO IDENTIFY THEIR PRIORITIES REGARDING 
AN ORGANIZATION S PROGRAM MIX, TO MAKE GENERAL OBSERVA- 
TIONS ON THE LEVEL OF RESOURCES DEEMED ADEQUATE TO CARRY 
OUT THE PROGRAMS, AND PERHAPS EVEN TO MAKE RECOMMENDATIONS 
WITH RESPECT TO PHASING DOWN LOWER PRIORITY ACTIVITIES. 

THE BEAUTY OF THE PROCESS IS THAT WHILE IT REQUIRES SOME 
SOUL-SEARCHING AND THE ASKING OF FUNDAMENTAL QUESTIONS 
BY EACH GOVERNMENT, IT DOES NOT NECESSARILY REQUIRE AN 
ANALYSIS OF PROGRAM INPUTS OR OUTPUTS. 

(2) TREND ANALYSIS - ANOTHER FORM OF PROGRAM REVIEW, 

TREND ANALYSIS, COMPLEMENTS AND GOES BEYOND THE PRIORITY 
RANKING METHOD BY EXAMINING THE HISTORICAL EVOLUTION OF 
PROGRAM INPUTS AND THEN EXTRAPOLATING INTO THE FUTURE THE 
APPROPRIATE DIRECTIONS AND RESOURCE LEVELS OF THE 

PROGRAMS. TREND ANALYSIS IDENTIFIES PROGRAM MOVEMENTS 
OR DIRECTION, AND TASKS THE ANALYST TO ASK IF THIS IS 
THE DIRECTION WHICH HIS GOVERNMENT BELIEVES TO BE THE 
MOST APPROPRIATE. THIS METHODOLOGY ENTAILS A FAIRLY 
RIGOROUS RE-EXAMINATION OF PAST PROGRAM AND BUDGET 
DOCUMENTS IN ORDER TO TRACE THE DEVELOPMENT OF PROGRAMS 
AND SUBPROGRAMS AND TO SEE IF THE OBIECTIVES ARE STATED 
DIFFERENTLY OR IF THE PROGRAM EMPHASIS HAS CHANGED. IT 
ALSO TAKES ACCOUNT OF OBJECT OF EXPENDITURE TRENDS. TREND 
ANALYSIS IS MORE OBJECTIVE THAN PRIORITY RANKING, BUT 
ITS INFERENCE THAT INPUTS ARE PROVIDING THE EXPECTED 
OUTPUTS IS NOT ALWAYS VALID . . . FOR EXAMPLE, THE NUMBER 
OF TEACHERS WITH GRADUATE DEGREES OR THE RATION OF L IBRARY 
BOOKS TO STUDENTS DOES NOT ALWAYS COINCIDE WITH THE AMOUNT 
OF KNOWLEDGE PASSED ON TO STUDENTS. HOWEVER, SHORT OF 
PERFORMANCE MONITORING OR PROGRAM EVALUATION, TREND 
ANALYSIS PROVIDES THE BEST WAY OF LOOKING AT A PROGRAM 
IN THE CONTEXT OF WHERE IT HAS BEEN, WHERE IT IS NOW, AND 
WHERE IT SHOULD BE GOING. 

(3) INPUT/OUTPUT ANALYSIS - THIS METHODOLOGY INVOLVES 
THE REVIEW OF PROGRAMS THROUGH THE USE OF COST ANALYSIS, 
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PERFORMANCE INDICATORS, AND OTHER MEASUREMENTS OF PROGRAM 
EFFECTIVENESS. IT REQUIRES A GOOD UNDERSTANDING OF 
SPECIFIC RESOURCE INPUTS AND THEIR RESPECTIVE OUTPUT 
CRITERIA. THUS, IT REQUIRES MUCH MORE AVAILABILITY TO 
DATA THAN CURRENTLY PROVIDED AND SIGNIFICANT MANPOWER 
REQUIREMENTS TO BE EFFECTIVE. THIS METHODOLOGY CAN AND 
SHOULD BE UTILIZED TO REVIEW PROGRAMS WHICH, THROUGH OTHER 
APPROACHES, HAVE BEEN IDENTIFIED AS POSSIBILY INEFFECTIVE 
OR OF MARGINAL BENEFIT. HOWEVER, THESE EVALUATIONS WILL 
OUT OF NECESSITY HAVE TO BE LIMITED IN NUMBER AND SCOPE. 
SEPARATE PAPERS ON THE TECHNIQUES OF PROGRAM EVALUATION 
ARE BEING DEVELOPED. 
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35. THE PROGRAM REVIEW PROCESS IN CAPITALS. 

36. THE GROUP OF EXPERTS MAY WISH TO MAKE SEVERAL GENERAL 
OBSERVATIONS ON THE SUBIECT OF INTERNAL REVIEW WITHIN 
GENEVA GROUP CAPITALS. 

37. FIRST, IT IS IMPORTANT FOR EACH GENEVA GROUP GOVERN- 
MENT TO DEVELOP THE INTERNAL CAPABILITY, OR IMPROVE UPON 
AN EXISTING CAPABILITY, TO ENGAGE IN A PROCESS OF REVIEW 
WHICH PROVIDES THAT GOVERNMENT'S TIMELY INPUT INTO THE 
PROCESS AGREED UPON BY THE GENEVA GROUP. THE UNITED 
STATES, FOR EXAMPLE, HAS RECENTLY ESTABLISHED WITHIN THE 
DEPARTMENT OF STATE A POLICY MANAGEMENT STAFF FOR INTER- 
NATIONAL ORGANIZATIONS, WHICH WILL BE RESPONSIBLE FOR 
DETERMINING U.S. POLICY OBIECTIVES IN EACH ORGANIZATION, 

RELATING THESE OBIECTIVES TO RESOURCE REQUIREMENTS, 
IDENTIFYING PROGRAMATIC AND INSTITUTIONAL CHANGES NECES- 
SARY FOR MORE EFFECTIVE AND EFFICIENT OPERATION, AND 
EVALUATING THE WORK OF THE ORGANIZATIONS IN RELATION TO 
UNCLASSIFIED 
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OBIECTIVES. THE REVIEW PROCESS DEVELOPED BY THIS STAFF 
WILL BE HELPFUL IN IDENTIFYING U.S. PRIORITIES AS WELL AS 
PINPOINTING PROGRAMS WHICH ARE OF MARGINAL BENEFIT OR 
OBSOLETE, AND WILL SERVE AS THE BASIS FOR DEVELOPING THE 
U.S. POSITIONS ON PROGRAMS WITHIN THE GENEVA GROUP. 

38. SECOND, IT IS IMPORTANT THAT THE INTERNAL REVIEW OF 
PROGRAMS IN CAPITALS RELY HEAVILY ON SUBSTANTIVE INPUTS 
FROM THE TECHNICAL EXPERTS IN THE RESPECTIVE MINISTRIES 
OR DEPARTMENTS. THE FOREIGN MINISTRIES ARE LIMITED IN 
THEIR SUBSTANTIVE CAPABILITIES TO ANALYZE PROGRAMS IN A 
WIDE VARIETY OF SECTORS, AND WOULD NOT BE ABLE TO RANK 
PROGRAM PRIORITIES WITH THE EASE OF THE TECHNICAL DEPART- 
MENTS. 

39. COORDINATION WITHIN THE GENEVA GROUP. 

40. THE REMAINING QUESTION TO BE ADDRESSED IS HOW THE 
RESULTS OF THE NATIONAL REVIEWS WILL BE PULLED TOGETHER 
INTO A GENEVA GROUP "POSITION" WHICH CAN BE PRESENTED TO 
THE SECRETARIATS. 

41 . MUCH OF THE COORDINATION AND DISCUSSION MUST BE 
CARRIED OUT AT THE GENEVA GROUP LOCAL LEVEL. THESE 
INDIVIDUALS ARE THE ONES MOST CLOSELY INVOLVED WITH THE 
ORGANIZATIONS AND CAN NEGOTIATE AMONG THEMSELVES MORE 
FREQUENTLY THAN THE OTHER GENEVA GROUP LEVELS. THE LOCAL 
GROUPS SHOULD PROVIDE FEEDBACK TO CAPITALS AND TO THE 
GENEVA GROUP GENERAL ON ISSUES RAISED DURING THESE DIS- 
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CUSSIONS. 

42. THE GENEVA GROUP GENERAL SHOULD BE THE FOCAL POINT 
OF THE PROCESS. RECOMMENDATIONS RECEIVED FROM THE LOCAL 
GROUPS SHOULD BE REVIEWED AND DISCUSSED BY THIS GROUP. 

SINCE THE CONSULTATIVE LEVEL MEETS ONLY TWICE A YEAR, ITS 
INVOLVEMENT WILL BE LITTLE MORE THAN APPROVING THE 
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RECOMMENDATIONS DEVELOPED AT THE LOCAL AND GENERAL LEVELS. 

43. IT WAS MENTIONED PREVIOUSLY THAT THE TASK OF COORDI- 
NATING THE VARIOUS REVIEWS MAY NOT BE DIFFICULT IF THE 
LEVEL OF PROGRAMMATIC DETAIL IS KEPT WITHIN REASON, 
ESPECIALLY DURING THE FIRST SEVERAL REVIEWS. PAST 
ATTEMPTS TO DEVELOP A CONSENSUS APPROACH TO SPECIFIC 
PROPOSALS AMONG GENEVA GROUP COUNTRIES HAVE MET WITH 
BOTH SUCCESS AND FAILURE. IT IS DIFFICULT TO SPECULATE ON 

THE PROSPECTS PRIOR TO THE FIRST ATTEMPT. HOWEVER, THE 
FAILURE TO REACH A CONSENSUS AMOUNG THE GROUP SHOULD NOT 
DETER GOVERNMENTS FROM PASSING THEIR RESPECTIVE RECOM 
MEDATIONS ON TO THE SECRETARIATS, EITHER UNILATERALLY OR 
IN CONJENCTION WITH OTHER GENEVA GROUP GOVERNMENTS. 

44. RECOMMENDATIONS TO IMPROVE SPECIALIZED AGENCY PLANNING 
AND PROGRAMMING. 

45. THE GROUP OF EXPERTS MAY WISH TO RECOMMEND SEVERAL 
ACTIONS WHICH WOULD IMPROVE THE PLANNING AND PROGRAMMING 
CAPABILITIES OF THE AGENCIES AND ENHANCE THE GENEVA GROUP 
REVIEW OF THESE PROGRAMS: 

(1) CLEARER SPECIFICATION OF PROGRAM OBJECTIVES - OFTEN- 
TIMES THE PROGRAM OBJECTIVES INCLUDED IN PROGRAM AND BUDGET 
DOCUMENTS ARE SO VAGUE AND NON-SPECIFIC THAT VIRTUALLY 
ANY PROGRAM COULD BE CONSTRUED AS FULFILLING, TO SOME 
DEGREE, THE STATED OBJECTIVE. ONCE OBJECTIVES ARE 
THGHTENED UP, MARGINAL PROGRAMS WILL BE EASIER TO IDENTIFY. 

(2) INCORPORATION OF REALISTIC EVALUATION CRITERIA INTO 
PROGRAM DEVELOPMENT - SPECIALIZED AGENCY PROGRAMS SHOULD 
INCLUDE IDENTIFIABLE EVALUATION CRITERIA AGAINST WHICH 

THE EFFECTIVENESS OF THE PROGRAMS CAN BE JUDGED. 
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(3) STRENGTHENING OF CURRENT EVALUATION CAPABILITIES OF 
THE SPECIALIZED AGENCIES - EVALUATION IS JUST BEGINNING 
TO CATCH ON IN THE AGENCIES. THEY SHOULD BE REQUESTED TO 
PLACE INCREASED EMPHASIS ON PROGRAM EVALUATION. 

(4) GREATER UTILIZATION OF THE JOINT INSPECTION UNIT - 
THE JIU SHOULD BE STRENGTHENED SO AS TO IMPROVE ITS 
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CAPABILITIES IN CARRYING OUT EVALUATIONS AND ITS ABILITY 
TO PROVIDE GUIDANCE TO THE AGENCIES IN DEVELOPING THEIR 
INTERNAL EVALUATION CAPABILITIES. UNQUOTE. 

46. FOR GENEVA: FOREGOING IS UNCLEARED PAPER FOR DIS- 
CUSSION BY GENEVA GROUP AT FEBRUARY 23-24 MEETING IF U.S. 

MISSION GENEVA DEEMS IT USEFUL. UNITED STATES DELEGATION 
SHOULD RESERVE RIGHT TO COMMENT POSITIVELY OR NEGATIVELY 
UPON PAPER AT THAT TIME OR SUBSEQUENTLY. VANCE 

UNCLASSIFIED 


« END OF DOCUMENT » 
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